Several systems currently exist in the commercial marketplace for tracking the movement of one or more objects within a limited area. Such systems are routinely used to follow the movement of human subjects for a range of purposes including medical analysis, input for lifelike animation, as well as sports measurement. The following companies provide machine vision-based motion analysis systems:                Motion Analysis Corporation and their HIRES 3D system;        Vicon with their Vicon 250 and 512 3D motion analysis systems;        Ariel Dynamics, Inc. with their APAS system;        Charnwood Dynamics with their CODA motion analysis system;        Peak Performance Inc. with their Motus system;        Biogesta with their SAGA—3 RT System;        Elite with their ELITEPlus Motion Analyser System;        Northern Digital with their Optotrak and Polaris systems, and        Qualisys with their ProReflex system.        Each of these systems, which are capable of working in real time and creating three-dimensional (3D) tracking information, employ a system of markers to be placed upon the object(s) to be tracked. The markers themselves are followed by an overlapping configuration of tracking cameras. As image information is analyzed from each camera's two-dimensional (2D) view, it is combined to create the 3D coordinates of each marker as that marker moves about in the designated tracking volume. Based upon the detected marker 3D locations as well as the pre-known relationship between the markers and the objects, each system is then able to “re-assemble” any given object's 3D movement. All of the systems share at least portions of the following common attributes:            1- All of the cameras that are used to view the marker and therefore object movement are pre-placed in fixed strategic locations designed to keep the entire tracking volume in view of two or more cameras.    2- Each camera is designed to capture a unique 2D view for a fixed portion of the tracking volume. The entire set of captured 2D information is combined by the system to create the 3D information concerning all markers and therefore objects.    3- For a marker to be located in the local (X, Y, Z) coordinate system during any given instant, it must be visible to at least two cameras within that instant.    4- They use a single tracking energy that is either from the visible spectrum (such as red light) or infrared (IR).    5- They add additional tracking energy in the form of LED-based ring lights attached to the tracking cameras.    6- They place special retroreflective markers on the objects to be tracked. These retroreflectors reflect a broad spectrum of energy including visible and IR light.    7- The markers do not comprise any special encoding and are most often identical in size and shape. Typical shapes are a rectangle, a circle, or a sphere.    8- They use the unique positional combination (i.e., the measured physical space relationship of the markers placed upon the object) to encode that object's identity. Hence, no two objects can have the same or a substantially similar positional combination of markers placed upon them. This “constellation” of markers covers the majority of the object's surface volume and as such requires that the entire volume remain substantially in view at all times.    9- They determine, confirm, or both determine and confirm the identity of each object simultaneously with the tracking of the objects as they move throughout the entire tracking volume.    10- After the cameras have been placed in their fixed positions, they calibrate the system prior to tracking by moving a special calibration tool throughout the combined views of all cameras. The calibration tool consists of two or more markers that are at a pre-known distance from each other. Once the calibration has been completed, none of the cameras may be moved before or during actual object tracking.
Each of these systems shares many drawbacks, especially when considered for use in a live sporting environment such as a sporting contest. Some of these drawbacks are as follows:    1- All of the tracking cameras must be set into fixed positions and then pre-calibrated prior to actual live tracking. This requirement precludes the use of automatic pan, tilt, and zoom cameras to collect additional information as directed by the system in anticipation of marker inclusions.    2- Each camera is positioned to have a unique and substantially perspective view of a given portion of the tracking volume. Each camera's perspective view contains a significant depth of field. Any given object traveling throughout this depth of field will be seen with substantially different resolutions depending upon whether the object is at the nearest or farthest point with respect to a given camera. Therefore, the system experiences a non-uniform resolution per object throughout the entire tracking volume. This non-uniform resolution affects at least the ease with which the system may be scaled up to cover larger and larger tracking volumes using a consistent camera arrangement.    3- If the tracking energy is red light, any human observers will also see the illuminated markers if they are within the narrow retroreflected cone of light.    4- If the tracking energy is red light, then the system is susceptible to reflections of red light caused by pre-existing lighting in the visible spectrum that may be reflected from red colored portions of the tracked objects or the tracking volume itself.    5- In order to reduce unwanted reflections when working with visible red light, the systems typically cover the objects in darker material and place black matting on the movement surface to help reduce unwanted reflections that become system noise. These techniques are not appropriate for a “live” environment.    6- If the tracking energy is IR, then the light sources employed only emit IR without any additional visible light. The additional visible light would normally act as an indicator to a human observer that the light is on, naturally causing them not to stare for prolonged periods. Continued exposure to any high-intensity energy including IR light can damage the retina of the eye.    7- When working with IR, these systems do not employ IR absorbent compounds to be placed upon the objects and tracking volume background surfaces before any markers are attached as a means of reducing unwanted reflections that become system noise.    8- Because the retroreflective markers work across a broad spectrum, they will reflect any visible energy, not just the chosen emitted tracking energy whether that be red light or IR. As such, they will for instance retroreflect any pre-existing lighting or portable lighting such as camera flashes that are typically used by human observers in a live environment.    9- Given that the preferred spherical markers do have an appreciable size, they are limited in the number of places that they can be placed upon the objects, especially in a live environment. Due to their size, they are impractical for use in live sporting contests especially contact sports such as ice hockey where they may become dislodged during normal play.    10- When the systems are used to track more and more objects, each with many markers, more and more instances arise when not all markers are in view of at least two cameras or in some cases in view of any camera. This is referred to as “inclusions” and also affects the ability of the system to accurately identify a given object since its identity is encoded in the unique “constellation” of the markers placed upon the object, the location of one or more of which is now unknown.    11- When the objects to be tracked are uniformed athletes such as ice hockey players versus non-uniformed human subjects, their body sizes and shapes become less distinguishable due to the standard pad sizes of their equipment and their loose-fitting jerseys. As body shapes become less distinguishable, then the unique “constellation” of markers used to identity a given uniformed athlete becomes less distinguishable and more markers must be added in order to clearly identify individual players.
The present inventors have addressed many of these drawbacks in their co-pending applications entitled:                Multiple Object Tracking System, application Ser. No. 09/197,219; Filed: Nov. 20, 1998 now U.S. Pat. No. 6,567,116        Method for Representing Real-Time Motion over the Internet, application Ser. No. 09/510,922; Filed: Feb. 22, 2000 now U.S. Pat. No. 6,707,487        Employing Electromagnetic By-Product Radiation for Object Tracking, application Ser. No. 09/881,430; Filed: Jun. 14, 2001 now abandoned        Visibly Transparent Wide Observation Angle Retroreflective Materials, application Ser. No. 09/911,043; Filed: Jul. 23, 2000        
Each of these patent applications is hereby incorporated by reference into the present application.
In these patents applications, the present inventors describe various aspects of a multiple object tracking system that functions in general to track many types of objects but that is especially constructed to track athletes during a live sporting event such as an ice hockey game. These patent applications teach at least the following novel components:    1- The system employs a matrix of separate overhead tracking cameras responsible for first locating any given object as a whole in a local (X, Y) area rather than in a (X, Y, Z) volume coordinate system. This technique yields a substantially uniform pixel resolution per area tracked providing a simple and regular approach to camera arrangement when the system is scaled to track larger areas.    2- The system employs separate sets of one or more pan, tilt, and zoom cameras per player to be tracked. These moveable cameras are automatically directed by the system based upon the (X, Y) location information that was first determined using the overhead tracking “area” cameras. Each of these volume cameras will collect (X, Y, Z) information from a particular view of the player to be combined with at least the (X, Y) information captured by the “area” cameras concerning the same player. Due to the system's ability to move and zoom each player-tracking camera, a substantially uniform pixel resolution per player is achieved. This technique provides a simple and regular approach to camera arrangement when the system is scaled to track more and more players.    3- The system preferably employs a non-visible tracking energy such as ultraviolet or infrared that is currently being generated by pre-existing lighting within the tracking area.    4- By using pre-existing lighting that is already in place with a purpose of illuminating the playing area for human observers, the system ensures that the observers will have a visible light indicator that the lamp is on. This will naturally keep the observer from staring at the lights and receiving an overexposure of non-visible tracking energy.    5- The system employs one or more reflective, retroreflective, fluorescent, and fluorescent retroreflective materials that are specifically designed to reflect only the chosen non-visible ultraviolet or infrared tracking energy and to be substantially transparent to visible light.    6- The system preferably employs markers that are made of ink which has minimal thickness and can be placed upon virtually any surface such as in the case of hockey a player's helmet, jersey, or gloves; the tape they use to wrap their stick; or the puck.    7- The system preferably encodes the player's unique identity into the markings placed exclusively upon the “top surface” of the player, such as the helmet or shoulders. In so doing, the player's identity can be determined solely from the (X, Y) area tracking cameras and is substantially unaffected by player “bunching” and subsequent body marker inclusions that primarily affect the view of the body below the helmet and shoulders.    8- The system preferably takes advantage of the reduced player movement and smaller area of the playing surface entrance and exit as well as the team benches in order to perform player identification. The unique characteristics of the entrance and exit and benches provides the opportunity to focus the overhead (X, Y) tracking cameras in a considerably smaller field-of-view such that the players' helmets and attached markings are considerably enlarged with respect to the entire captured image. This in turn ensures that the space available for a marking on the helmet is sufficient to completely encode and therefore identify a given player through the use of more complex symbol patterns similar to bar codes. As previously mentioned, since the unique player code is therefore fully contained on the helmet, only the overhead (X, Y) cameras are necessary to determine identity thereby eliminating the effect of body marker inclusions caused when the (X, Y, Z) cameras' fields-of-view are blocked.    9- By separating entrance and exit and bench tracking and identification from playing surface tracking, it is possible to place a multi-frequency responsive marker at least on the player's helmet. For instance, the complex symbol patterns used to encode the player's identity can be created with an UV ink while the helmet tracking mark can be created with an IR ink, or vice versa. This switching of frequencies effectively doubles the available marking area of at least the helmet and potentially any other “top surface” such as the shoulders.    10- The area cameras have mutually exclusive fields-of-view with slight edge-to-edge overlap for calibration purposes. This calibration process is performed prior to live tracking.    11- The volume cameras are first calibrated with respect to their pan, tilt, and zoom drive mechanisms also prior to actual tracking. Their field-of-view will constantly overlap one or more area cameras. The combination of this overlapping area and volume information is then used by the system for dynamic re-calibration and adjustment of the volume cameras. The system thereby permits individual cameras to move and be recalibrated simultaneously with actual tracking.    12- The system employs absorber compounds that are to be placed upon the objects and playing surface prior to placing the markers in order to cut down or eliminate unwanted reflections that are system noise.    13- The system employs predictive techniques based upon the object's last known position, acceleration, velocity, and direction of travel to minimize the search time required to locate the object in subsequent video frames.
Also currently existing in the commercial marketplace are the following important components:    1- Wide-angle retroreflectors are capable of reflecting light in a wider cone than typical retroreflectors. These are available in both cube cornered and microscopic bead optical body formats. They provide the opportunity to move the lighting source further away from the tracking cameras when using retroreflective materials.    2- Fluorescent laser dyes are capable of absorbing visible light just below the 700 nm wavelengths that are still visible and converting it into IR light just above the 700 nm region that is non-visible. When using IR, these dyes provide the opportunity to convert visible energy as emitted by pre-existing arena lighting into IR tracking energy thus eliminating the need to add lighting that specifically radiates IR into the tracking volume.    3- Fluorescent laser dyes are capable of absorbing UV light around 330 nm wavelengths and converting it into UV light around 390 nm. This conversion is important given that certain commercially available low-cost digital imaging cameras are designed to have a higher responsivity to UV light especially around the 390 nm range. Furthermore, existing arena lighting such as Metal Halide Lamps currently generate UV energy in the frequency range of 315 to 400 nm. By absorbing the shorter wavelength UV energy around 330 nm and then radiating additional UV energy around 390 nm, the fluorescent dyes will essentially “double up” on the preferred narrow band of tracking frequencies.    4- Notch filters may be used with the tracking cameras and are capable of passing very narrow bands of specific frequencies of energy. This provides the opportunity to place reflective, fluorescent, or retroreflective materials that operate at different frequency ranges onto different players to assist in the identification process.
The present inventors have described in their co-pending applications many useful component and system solutions to the problems that are inherent within the existing systems. Additional components as described above also exist. It is possible to create several different and yet effective machine vision systems for tracking multiple moving objects based upon the novel components disclosed within the present application and four co-pending applications. What is needed is an understanding of how all of these teachings can be combined to form several different machine vision systems, each with their own novel optimizations.
In addition to the aforementioned machine vision system solutions to multi-object tracking it should be noted that at least two other companies are attempting to provide systems for similar purposes. Both Trackus, a Massachusetts-based company, and Orad, an Israeli-based company, are attempting to develop real-time “beacon”—based tracking technology for sporting events. Orad has produced a working system to follow horse racing; Trakus is the only company currently attempting to follow players in an ice hockey game. While Orad's solution is essentially similar at the highest levels, the technology will be explained based upon information gathered concerning Trakus.
Trakus' solution includes a microwave based transmitter and receiver that will track a single point within the helmet of each player. There are many deficiencies with this proposed solution as compared to machine vision systems in general and the novel teachings of the present inventors in particular. One of the most important distinctions is the “active” and potential harmful nature of the microwave technology. If used for tracking youth sports, it is anticipated that the average parent would balk at the idea of placing even a low-power microwave device into the helmet of their child. Furthermore, there are significant reflection problems due to the hard interior surfaces of a hockey arena that must be resolved before this technology can effectively track even a single point (the helmet) on every player on both teams. As already discussed, machine vision-based systems employ “passive” markers that are capable of tracking 14 or more points (the head and every major joint) on every player in real time. The present invention furthermore uniquely teaches a system that can also track game equipment and the puck, devices that have surfaces that cannot be substantially altered by the normal size of traditional markers.